View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001004K18X001.00 SKB SWANSWpublic2020-11-02 16:152020-12-11 21:55
Reporter(ALTech) Wooshin Kang Assigned To(ALTech) Wooshin Kang Due Date2020-11-06 17:10
PriorityhighSeveritys2-severeReproducibilityalways
Status closedResolutionfixed 
Summary0001004: [SWAN] Selection audio source for NUGU app when user press nugu button rcu.
DescriptionWhen user press "nugu" button, audio source for nugu app should be selected to bt-rcu from 4 ch mic.

Thanks.
TagsNo tags attached.
Attach Tags

Users monitoring this issue

User List (SW) Jacky Chiang , (SW) Jim Lin , (SW) Kerwin Chen

Activities

(SW) River Wong

2020-11-03 10:23

developer   ~0004828

Hi Wooshin,

Do you mean we do not need to capture audio from RCU and just capture audio from Far Field MIc when RCU NUGU button is pressed?

(ALTech) Wooshin Kang

2020-11-03 11:54

developer   ~0004833

Hi River,

You need to implement dynamic change from BT RCU to 4 ch mic or from 4ch mic to BT RCU.

Thanks.

(SW) River Wong

2020-11-03 13:58

developer   ~0004838

Last edited: 2020-11-03 13:59

 Hi Wooshin,

We do no understand what do you mean of "dynamic change". It should depends on some condition to capture audio from different audio device (For example, we need a condition to check to decide if we need to get audio from RCU or Far Field MIC).
So,please give detail description or document about your request.

PS: I list some background of NUGU RCU implementation for your reference.
NUGU APP use standard API AudioRecord(VoiceRecogniztion) to get the audio data from android framework.
In Android Framework, we modify code to identify this API is from NUGU APP or Google Assistant App(Google Assistant uses the same API with same parameters) to call Audio HAL with different input Audio device as parameter.
In Android Audio HAL, we check the input device parameter. If it is for NUGU audio record, we open the RCU audio device to get audio data from BT RCU.

(SW) Jacky Chiang

2020-11-03 16:12

manager   ~0004842

Last edited: 2020-11-03 16:17

 Based on the implementation in Android framework to support Google Assistant and Nugu, the Nugu App shall control the switch and determine if it needs to read data from BT RCU or 4ch mic for FFV when user press Nugu(Nugu app would receive key event for this) but audio HAL in framework is in passive role.

(ALTech) Wooshin Kang

2020-11-03 16:36

developer   ~0004843

Last edited: 2020-11-03 16:39

 Dear River

I can only explain to you like below, I have no source code. There are many differences in understanding the problem with and without source code. Please understand me.

Normally Nugu app get voice input from 4 mic, but if user wants to use RCU for voice input, user has to press "nugu button" of RCU. This time you have to change voice input path to RCU.

My thinking of NUGU APP's behavior is following after discussion with skb manager and engineer of INTEK.
1) user press nugu button on rcu.
-> we have to remember previous input key was nugu button
2) nugu app will open audiorecord class for getting voice.
-> select RCU for voice input.
3) After getting voice, nugu app will close audiorecord class.
-> select 4 mic for voice input.

If you have any concern about upper steps, Please inform me.

(ALTech) Wooshin Kang

2020-11-09 10:52

developer   ~0004898

Hi river,

I have discussed with skb and skt manager about.
If we make new audio source type for rcu voice, they will control.

Internal 4 channel mic : MIC
RCU MIC : VOICE_RECOGNITION

Please refer for mic type.
https://developer.android.com/reference/android/media/MediaRecorder.AudioSource#MIC

If you have any problem for applying upper, please inform me.

Thanks.

(SW) River Wong

2020-11-09 13:38

developer   ~0004900

Hi Wooshin,

Thanks. It is more reasonable to use new type for FFV.
We will based on this to implement. If we encounter any problem, then we will inform you.

(ALTech) Wooshin Kang

2020-11-11 22:34

developer   ~0004952

Hi River,

Please change priority of this issue to more higher, current SKT managers are implementing NUGU APP, so we need this function ASAP.

Thanks.

(SW) Jacky Chiang

2020-11-12 14:50

manager   ~0004963

Hi Wooshin,

    I checked with our engineer and we are working on this to support MIC(internal 4 channel mic) in audio HAL.
We put this issue in higher priority.

Thanks ~

(ALTech) Wooshin Kang

2020-11-16 14:53

developer   ~0005007

HI all,

we have to finish it before QA. QA will be started 23th Nov.

(SW) River Wong

2020-11-17 15:06

developer   ~0005022

Hi Wooshin,

About getting FFV 4 MIC by using AudioRecord(MIC,...), we find a problem in current last code.
The "MIC" is mapping to "AUDIO_DEVICE_IN_BUILTIN_MIC" in framework and pass to Audio HAL as a "devices" parameter.
In Audio HAL, we will check the "devices" parameter, if it is "AUDIO_DEVICE_IN_BUILTIN_MIC", then we know APP is using AudioRecord(MIC) to ask us to capture FFV 4 MIC audio data.
But form recently SK committed a code, they seem also use "AUDIO_DEVICE_IN_BUILTIN_MIC" to do NUGU preprocessing. You can refer to following code:

SK(sungjun.bhang@sk.com) committed a code in vendor/synaptics/vsxxxaudio/AudioStreamIn.cpp to check AUDIO_DEVICE_IN_BUILTIN_MIC:
==================================================================================================
 #if 1 // SKT_NUGU_PREPROCESS
         if (devices == AUDIO_DEVICE_IN_BUILTIN_MIC || devices == AUDIO_DEVICE_IN_DEFAULT) {
             MVLOGD("Open AudioStreamNUGUPPIn!!");
             audioStreamIn = new AudioStreamNUGUPPIn(handle, config, devices);
         } else {
             MVLOGD("unsupported devices: %u", devices);
         }
 #else
=================================================================================================

Then it will have some conflict if we and SK use the same device type "AUDIO_DEVICE_IN_BUILTIN_MIC".
I think we need to use different device type with them.
Could you help to check this with them?
For example, do they also use AudioRecord(MIC) to get NUGU preprocessing audio data? If so, will it be closed when NUGU APP uses AudioRecord(MIC) ?

(SW) Kerwin Chen

2020-11-17 18:16

developer   ~0005026

Hi Wooshin,

Please help to check with SKT how they use AudioRecord for internal 4 channel MICs and RCU voice.
We use latest NUGU release (3.5.17-user-201110) on BitBucket, NUGU doesn't use "MIC" as source type for internal 4 channel MICs.
This change will have impact not only on RCU NUGU but also Google Assistant, since Google Assistant also use "VOICE_RECOGNITION".
Now, we have to use previous release of NUGU to test RCU voice.
Moreover, if it is not corrected, we can't commit changes to BitBucket and release FW for testing.

(ALTech) Wooshin Kang

2020-11-17 20:02

developer   ~0005028

Hi River and Kerwin,

I will discuss it in tommorow meeting with SK and share to you.

Thanks.

(ALTech) Wooshin Kang

2020-11-18 08:26

developer   ~0005031

Hi River and Kerwin,

What can we use instead of MIC and VOICE_RECOGNITION ?
Please explain about other option.

Thanks.

(SW) Kerwin Chen

2020-11-18 08:37

developer   ~0005032

Hi Mr. Wooshin Kang,

For source type, we suggest to keep the same as your post in https://172.18.223.170/vaas/view.php?id=1004#c4898.
Internal 4 channel mic : MIC
RCU MIC : VOICE_RECOGNITION

In this way, Google Assistant and RCU MIC will use the same source type.
We will check caller thread to distinguish Google Assistant and RCU MIC. (This is the same mechanism as Smart3)
Thanks !

(ALTech) Wooshin Kang

2020-11-18 10:51

developer   ~0005037

Hi kerwin,

I have attached nugu app which use source selection.

thanks.
nugu-stb-bfx_ua300-user-3.5.17-audiosource-test.apk (26,373,047 bytes)

(SW) River Wong

2020-11-18 14:00

developer   ~0005043

Hi Wooshin,

For more clearly describe our problem, I draw a SW stack flow about AudioRecord() in attached ppt file.
As you can see from the ppt, the NUGU ProProcess App should use a different "AudioSrouce" that can not be "MIC" and "VOICE_RECOGNITION".
Currently, we think it use "VOICE_RECOGNITION" and this will cause conflict.
There is an new AudioSource "ECHO_REFERENCE" which is mapping to "AUDIO_DEVICE_IN_ECHO_REFERENCE". Maybe SK can use it.

BTW, what is the purpose of test apk? Is it able to configure the AudioSource? Please advice how to use it.
AudioRecord_issue_on_SWAN.pptx (66,151 bytes)

(ALTech) Wooshin Kang

2020-11-18 17:05

developer   ~0005044

Hi River,

"nugu-stb-bfx_ua300-user-3.5.17-audiosource-test.apk " is test app which select audio input to MIC or VOICE_RECOGNITION.
This is requested by River. please check comment of River.

(SW) River Wong

2020-11-19 13:47

developer   ~0005061

Hi Wooshin,

As you know, Kerwin already committed the code for capturing NUGU RCU audio when NUGU APP uses AudioRecord("VOICE_RECOGNITION").
But regarding capturing 4 FFV MIC audio for NUGU FFV, could you help to confirm with SK if they will handle it by themselves and we do not need to do anything about it?

(SW) River Wong

2020-11-23 09:02

developer   ~0005097

Hi Wooshin,

Any update for confirming the apturing 4 FFV MIC audio for NUGU FFV?
Could you help to confirm with SK if they will handle it by themselves and we do not need to do anything about it?

(ALTech) Wooshin Kang

2020-11-23 09:28

developer   ~0005098

Hi, River
 
Last Friday was holiday for SK Group.
I will check status and will share to you about result.

Thanks.

(ALTech) Wooshin Kang

2020-11-26 09:35

developer   ~0005163

Dear River,

In the test situation of manager Seung-jun Bang below, it seems that when BT is connected, it is automatically detected as Wired Headset regardless of the type designation.
When there are two available MICs, it will be necessary to check whether it is automatically connected to the BT side.

Please check it.

Following is mail from SKT.
================================================================================================
Hello, I am Mr Bang of SKT

When i check AvailableInputDevice, It is always wired_headset like following

Device 5:
  - id: 25
  - tag name: BMM_BA02_C67
  - type: AUDIO_DEVICE_IN_WIRED_HEADSET
  - address: 40:19:20:20:7C:67
  - name: BMM_BA02_C67
  - Profiles:
      Profile 0:
          - format: AUDIO_FORMAT_PCM_16_BIT
          - sampling rates:8000, 16000, 32000, 48000
          - channel masks:0x000c, 0x0010

I
In this case, it seems that this device will always be selected according to the input device priority.
Thanks.
================================================================================================

(SW) Kerwin Chen

2020-11-26 15:50

developer   ~0005178

Hi Mr. Wooshin Kang,

We change the order in Android framework today.
Now, the priority of BUILTIN_MIC is higher that WIRED_HEADSET if application uses 'MIC' as audio source for AudioRecord.
Please sync sources with BitBucket and test again.
Thanks !

(ALTech) Wooshin Kang

2020-11-26 20:03

developer   ~0005185

Hi kerwin,

Following is opinion of SKT manager.

Please check also "AUDIO_SOURCE_VOICE_COMMUNICATION". It is used by nugu call.

Thanks.

(SW) Kerwin Chen

2020-11-26 20:27

developer   ~0005188

Hi Mr. Wooshin Kang,

The change for NUGU CALL is committed to BitBucket.
Thanks !

(ALTech) Wooshin Kang

2020-11-30 08:57

developer   ~0005202

Hi Kerwin,

In my box, it is working correctly but other box is not working.
Version of all box is same. we are finding root cause why all box do not same behavior.
If update, i will share.

Thanks.

(SW) River Wong

2020-11-30 13:39

developer   ~0005214

HI Wooshin,

In above comment 0001004:0005098, you said you will help to confirm with SKT about:
"Capturing 4 FFV MIC audio for NUGU FFV.
Could you help to confirm with SK if they will handle it by themselves and we do not need to do anything about it?"
Do you confirm it ?

(ALTech) Wooshin Kang

2020-12-11 21:55

developer   ~0005433

It is working now.

Issue History

Date Modified Username Field Change
2020-11-02 16:15 (ALTech) Wooshin Kang New Issue
2020-11-02 16:15 (ALTech) Wooshin Kang Status new => assigned
2020-11-02 16:15 (ALTech) Wooshin Kang Assigned To => (SW) River Wong
2020-11-03 10:21 (SW) River Wong Issue Monitored: (SW) Jacky Chiang
2020-11-03 10:21 (SW) River Wong Issue Monitored: (SW) Kerwin Chen
2020-11-03 10:23 (SW) River Wong Note Added: 0004828
2020-11-03 11:54 (ALTech) Wooshin Kang Note Added: 0004833
2020-11-03 13:58 (SW) River Wong Note Added: 0004838
2020-11-03 13:59 (SW) River Wong Note Edited: 0004838
2020-11-03 16:12 (SW) Jacky Chiang Note Added: 0004842
2020-11-03 16:16 (SW) Jacky Chiang Note Edited: 0004842
2020-11-03 16:17 (SW) Jacky Chiang Note Edited: 0004842
2020-11-03 16:36 (ALTech) Wooshin Kang Note Added: 0004843
2020-11-03 16:39 (ALTech) Wooshin Kang Note Edited: 0004843
2020-11-09 10:52 (ALTech) Wooshin Kang Note Added: 0004898
2020-11-09 13:38 (SW) River Wong Note Added: 0004900
2020-11-11 22:34 (ALTech) Wooshin Kang Note Added: 0004952
2020-11-12 14:50 (SW) Jacky Chiang Note Added: 0004963
2020-11-13 11:05 (SW) Jacky Chiang Project T99X171.00 SKB Eagle => K18X001.00 SKB SWAN
2020-11-16 14:53 (ALTech) Wooshin Kang Note Added: 0005007
2020-11-16 15:27 (SW) Jacky Chiang Issue Monitored: (SW) Jim Lin
2020-11-17 15:06 (SW) River Wong Note Added: 0005022
2020-11-17 18:16 (SW) Kerwin Chen Note Added: 0005026
2020-11-17 20:02 (ALTech) Wooshin Kang Note Added: 0005028
2020-11-18 08:26 (ALTech) Wooshin Kang Note Added: 0005031
2020-11-18 08:37 (SW) Kerwin Chen Note Added: 0005032
2020-11-18 10:51 (ALTech) Wooshin Kang File Added: nugu-stb-bfx_ua300-user-3.5.17-audiosource-test.apk
2020-11-18 10:51 (ALTech) Wooshin Kang Note Added: 0005037
2020-11-18 14:00 (SW) River Wong File Added: AudioRecord_issue_on_SWAN.pptx
2020-11-18 14:00 (SW) River Wong Note Added: 0005043
2020-11-18 17:05 (ALTech) Wooshin Kang Note Added: 0005044
2020-11-19 13:47 (SW) River Wong Note Added: 0005061
2020-11-23 09:02 (SW) River Wong Note Added: 0005097
2020-11-23 09:03 (SW) River Wong Assigned To (SW) River Wong => (ALTech) Wooshin Kang
2020-11-23 09:28 (ALTech) Wooshin Kang Note Added: 0005098
2020-11-26 09:35 (ALTech) Wooshin Kang Note Added: 0005163
2020-11-26 15:50 (SW) Kerwin Chen Note Added: 0005178
2020-11-26 20:03 (ALTech) Wooshin Kang Note Added: 0005185
2020-11-26 20:27 (SW) Kerwin Chen Note Added: 0005188
2020-11-30 08:57 (ALTech) Wooshin Kang Note Added: 0005202
2020-11-30 13:39 (SW) River Wong Note Added: 0005214
2020-12-11 21:55 (ALTech) Wooshin Kang Status assigned => closed
2020-12-11 21:55 (ALTech) Wooshin Kang Resolution open => fixed
2020-12-11 21:55 (ALTech) Wooshin Kang Note Added: 0005433